Aiming at the problems of large model parameters, high computational complexity and low accuracy of traditional violence detection methods, a method of violence detection in video based on temporal attention mechanism and EfficientNet was proposed. Firstly, the foreground image obtained by preprocessing the dataset was input to the network model to extract the video features, meanwhile, the frame-level spatial features of violence were extracted by using the lightweight EfficientNet, and the global spatial-temporal features of the video sequence were further extracted by using the Convolutional Long Short-Term Memory (ConvLSTM) network. Then, combined with temporal attention mechanism, the video-level feature representations were obtained. Finally, the video-level feature representations were mapped to the classification space, and the Softmax classifier was used to classify the video violence and output the detection results, realizing the violence detection of video. Experimental results show that the proposed method can decrease the number of model parameters, reduce the computational complexity, increase the accuracy of violence detection and improve the comprehensive performance of the model with limited resources.